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ABSTRACT 


The  purpose  of  this  thesis  is  to  analyze  the  feasibility 
of  accrediting  the  Janus  combat  simulation  for  use  with  an 
Army  concept  called  Model-Test-Model.  Aircraft  detection 
ranges  from  an  operational  field  test  are  compared  to  similar 
ranges  generated  by  the  Janus  combat  simulation.  Means  and 
distributions  of  detection  ranges  for  both  helicopter  and 
fixed-wing  aircraft  are  studied.  Methods  of  collecting 
detection  data  in  field  tests  and  representing  aircraft  in 
Janus  must  be  improved  for  Janus  to  be  accredited  for  aircraft 
detection  range  analysis  of  field  tests. 
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I.  INTRODUCTION 


A.  THESIS  OBJECTIVE 

The  objective  of  this  thesis  is  to  validate  or  accredit 
the  Janus  combat  simulation  system  for  ground  to  air 
detections  within  the  post-test  phase  of  the  Model-Test-Model 
(MTM)  concept.  The  approach  will  be  to  compare  field  test 
detection  range  data  of  the  Line  of  Sight-Forward-Heavy 
surface-to-air  missile  system  with  that  of  the  Janus  combat 
simulation. 

B.  DESCRIPTION  OF  THE  LINE  OF  SIGHT-FORWARD-HEAVY  (LOS-F-H) 

SYSTEM 

For  years,  the  Army  has  recognized  the  need  for  a  capable 
low-altitude,  short-range  air  defense  weapon  system  to  replace 
tne  outdated  Chapparal  and  Vulcan  systems.  The  LOS-F-H  should 
eventually  fill  that  need. 

The  LOS-F-H  is  a  surface-to-air  system  mounted  on  a 
modified  Bradley  fighting  vehicle.  The  only  major  armament  of 
the  LOS-F-H  is  its  missiles.  The  LOS-F-H  carries  eight 
missiles  onboard.  It  is  manned  by  a  crew  of  three:  a 
commander/radar  operator  (RO) ,  a  gunner/electro-optics  (EO) 
operator,  and  a  driver.  The  LOS-F-H  has  its  own  acquisition 
and  tracking  radar  which  can  track  on  several  aircraft  while 
scanning  for  others  out  to  a  range  of  twenty  kilometers.  The 
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crew  uses  a  Forward  Looking  Infrared  (FLIR)  sensor  or  an 
optical  (TV)  sensor  to  track  and  help  identify  the  target. 
Identification  of  the  target  is  finalized  using  the 
Identification,  Friend,  or  Foe  (IFF)  device,  which  receives  an 
encrypted  transmission  sent  out  by  friendly  aircraft  [Ref.  1] . 

The  LOS-F-H  missile  is  designated  as  a  'laser  beam  rider' 
because  it  is  guided  by  a  coded  laser  beam  from  the  fire  unit. 
The  laser  beam  rider  capability  lessens  the  effectiveness  of 
the  enemy's  use  of  countermeasures  to  disrupt  missile  flight 
through  radar  jamming. 

The  LOS-F-H  weapon  system  weighs  approximately  31  tons 
fully  loaded,  including  eight  112  pound  missiles  in  ready-to- 
fire  canisters.  Figure  1  is  a  picture  of  the  LOS-F-H  and  the 
crew  positions. 
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C.  DEFINITION  OF  DETECTION 


For  the  Janus  trials,  aircraft  detection  occurs  when  the 
LOS-F-H  radar  determines  that  some  type  of  airframe  is 
approaching,  records  its  location,  and  identifies  the  target 
as  friendly  or  hostile.  The  field  test  trials  require  a 
further  step  for  aircraft  detection;  a  crewman  must  be  aware 
of  the  approaching  target.  The  differences  in  these  two 
processes  make  their  comparison  difficult.  Though  hostile  and 
friendly  aircraft  detections  were  recorded  in  both  the  field 
test  and  Janus,  our  concern  will  be  with  detections  of  hostile 
aircraft  only. 

D.  MODEL-TEST-MODEL  CONCEPT 

Field  tests  are  fundamental  to  the  acquisition  of  any  new 
weapon  system.  Critical  to  the  quality  of  a  field  test  is 
that  it  accurately  portrays  the  capabilities  of  the  system 
within  its  operational  context.  One  method  that  will  help 
improve  the  quality  of  field  tests  is  Mode 1 -Test-Mode 1 . 

The  term  Model-Test-Model  refers  to  the  three  primary 
phases:  the  pre-test  phase,  the  field  test,  and  the  post-test 
phase. 

1 .  PRE-TEST  PHASE 

The  pre-test  phase  involves  use  of  a  combat  simulation 
model  by  members  of  the  field  test  evaluation  team,  the  Army 
proponent,  the  modellers,  and  maneuver  unit  leaders.  During 
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this  phase,  scenarios  are  developed  within  the  simulation  that 
can  be  used  during  the  field  test.  The  scenarios  must 
incorporate  the  restrictions  placed  on  the  test,  including 
representation  of  the  terrain,  numbers  of  players,  and  the 
force  mix.  The  maneuver  unit  leaders  try  to  determine  tactics 
for  each  scenario,  such  as  routes  of  march  and  defensive 
positions.  The  results  of  the  scenario  development  are  used 
by  the  field  test  evaluation  team  in  developing  and  improving 
their  data  collection  and  analysis  methods  and  to  improve  the 
conduct  of  the  field  test.  Tactics  that  were  successful  in 
both  the  model  and  field  test  should  be  provided  to  the  Army 
proponent  to  update  doctrine. 

The  pre-test  phase  also  can  provide  training  for  the 
maneuver  unit  leaders.  Many  past  tests  have  indicated  that 
several  trials  in  a  field  test  may  be  necessary  for  player 
units  to  get  adequately  organized  and  trained.  Exposure  to 
the  scenario,  terrain,  and  systems  that  will  be  involved  in 
the  field  test  during  the  pre-test  phase  may  make  the  earlier 
field  trials  better,  since  the  maneuver  leaders  are  presumably 
familiar  with  major  facets  of  the  field  test  from  the  outset. 

2.  FIELD  TEST  PHASE 

During  the  field  test  phase,  the  system  is  tested  under 
realistic  combat  conditions  to  determine  its  operational 
effectiveness  and  suitability.  It  is  important  that  typical 
users  of  this  system  are  used  in  the  field  test  and  that  the 
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threat  forces  employed  are  realistic.  The  specified  minimum 
acceptable  operational  requirements  delineated  by  the  system's 
proponent  must  be  adequately  tested. 

The  use  of  MTM  in  the  pre-test  phase  allows  the  field 
testers  to  use  improved  scenarios  derived  from  pre-test 
modelling.  The  amount  of  replication  of  the  pre-test  tactics 
employed  may  differ  for  force  development  tests  and 
operational  tests. 

Force  development  tests  are  conducted  early  in  the  weapon 
acquisition  process  and  are  meant  to  examine  the  effectiveness 
of  existing  or  proposed  concepts  of  training,  doctrine, 
organization,  and  material.  Therefore,  force  development 
tests  should  include  trials  which  attempt  to  replicate  as 
closely  as  possible  some  of  the  tactics  developed  by  the 
maneuver  unit  leaders  during  the  pre-test  modelling.  This 
provides  field  test  results  for  some  of  the  tactics  examined 
in  the  simulation  and  helps  determine  if  these  successes  can 
be  approximated  in  the  testing  environment. 

Operational  tests,  both  initial  and  follow-on,  occur  later 
in  the  acquisition  process  and  are  less  controlled  than  force 
development  tests.  They  focus  on  the  operational 
effectiveness  of  new  systems  and  whether  they  are  worthy  of 
production.  Operational  tests  differ  from  force  development 
tests  in  that  tactical  realism  is  critical  and  ground 
commanders  do  not  follow  a  'script'.  Rather,  maneuver  leaders 
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use  their  own  judgements  in  an  operational  test,  with  the  aid 
of  the  knowledge  gained  from  the  pre-test  phase. 

3 .  POST-TEST  PHASE 

Following  the  field  test,  the  post-test  phase  begins.  The 
immediate  goal  of  the  post-test  phase  is  to  validate  or 
accredit  the  simulation  model.  Since  we  are  primarily 
concerned  with  post-test  model  accreditation,  the  process  for 
accomplishing  this  goal  is  detailed  in  this  thesis. 

As  a  second  goal  of  this  phase,  evaluators  can  use  the 
improved  simulation  for  cautious  predictions.  Constraints  of 
the  field  test,  such  as  numbers  of  players,  dimensions  of  the 
maneuver  areas,  and  types  of  terrain,  could  be  altered  within 
the  simulation.  Results  of  runs  with  the  modified  simulation 
model  could  add  valuable  information  for  consideration  by  the 
evaluator. 

MTM  combines  force-on-force  battle  simulation  and  field 
testing  with  the  goal  of  improving  them  both.  MTM  is  used  to 
support  test  design  and  evaluation  of  test  results. 

4.  DEFINITION  OF  VALIDATION  AND  ACCREDITATION 

Before  a  model  can  be  used  to  represent  an  actual  field 
test,  it  must  be  shown  that  the  model  is  a  measureably  close 
representation  of  that  field  test.  The  degree  to  which  it  can 
be  shown  to  represent  the  field  test  will  determine  whether 
the  model  is  validated  or  accredited  [Ref.  2]. 

The  concepts  of  model  validation  and  accreditation  were 
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addressed  by  Walter  Hollis,  the  Deputy  Undersecretary  of 
Defense  for  Operations  Research.  He  defines  model  validation 
as  follows. 

The  process  of  determining  that  a  model  is  an  accurate 
representation  of  the  intended  real-world  entity  from  the 
perspective  of  the  intended  use  of  the  model  [Ref.  3]. 

In  this  thesis,  our  intent  is  not  to  attempt  to  show  that  the 

model  is  an  accurate  representation  of  the  LOS-F-H  field  test, 

as  we  are  concerned  only  with  analyses  of  aircraft  detection 

ranges. 

On  the  other  hand,  accreditation  is  concerned  with  more 
specific  applications.  Mr.  Hollis  defines  accreditation  as 
follows. 

Certification  that  a  model  is  acceptable  for  use  for  a 
specific  type  of  application.  Accreditation  is  possible 
even  if  the  model  is  not  fully  validated  [Ref.  3]. 

Determining  that  a  model  accurately  represents  all  aspects 
of  a  field  test  may  be  difficult.  Accreditation,  or 
demonstrating  that  a  model  is  accceptable  for  certain 
applications,  such  as  a  comparison  of  detection  ranges,  might 
be  feasible. 

E.  HOW  THIS  THESIS  FITS  INTO  MODEL-TEST-MODEL 

In  this  thesis,  we  are  concerned  strictly  with 
accreditation  of  the  Janus  model.  Specifically,  we  compare 
aircraft  detections  by  the  Line  of  Sight-Forward-Heavy  (LOS-F- 
H)  from  an  actual  field  test  conducted  in  May  and  June  of  1990 
with  simulations  of  the  selected  test  trials  with  the  Janus 
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model.  Only  detections  by  LOS-F-H  systems  of  fixed  wing 
aircraft  and  helicopters  are  analyzed.  In  the  next  two 
chapters  we  discuss  the  LOS-F-H  field  test  and  the  Janus 
simulation  model. 


8 


II.  CONDUCT  AND  DESCRIPTION  OF  THE  FIELD  TEST 


A.  DESCRIPTION  OF  THE  FIELD  TEST 

1.  FIELD  TEST  BACKGROUND 

The  purpose  of  the  field  test  was  to  evaluate  the 
operational  effectiveness  and  suitability  of  the  LOS-F-H 
platoon  in  performing  its  designated  role  of  providing  air 
defense  for  heavy  maneuver  forces  in  the  forward  area. 

The  agency  conducting  the  field  test  was  TRADOC  Test  and 
Experimentation  Command,  Experimentation  Center  (TEC) .  The 
test  was  an  Initial  Operational  Test  and  Evaluation  (lOTE) . 
For  the  sake  of  simplicity,  we  will  refer  to  it  as  the  'field 
test'  throughout  this  document. 

Over  1000  personnel  participated  in  the  field  test,  and 
almost  600  of  them  were  members  of  tactical  units  from  the 
Army,  Air  Force,  and  the  Air  National  Guard.  The  field  test 
took  place  at  Fort  Hunter  Liggett,  California  in  the  spring 
of  1990. 

2.  DESCRIPTION  OF  INSTRUMENTATION 

All  ground  vehicles  and  aircraft  had  sophisticated 
instrumentation  packages  installed.  This  equipment  consisted 
of  laser  sensors  and  emitters  which  were  intended  to  provide 
updates  of  battle  progress.  This  capability  is  called  real¬ 
time  casualty  assessment  (RTCA) .  With  this  instrumentation  on 
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each  vehicle,  player  locations,  as  well  as  important  events 
such  as  engagements,  could  be  monitored  and  recorded.  Much 
data  gathering  and  analysis  depended  on  the  accuracy  of  this 
process . 

3.  FIELD  TEST  PHASES 

The  field  test  consisted  of  three  phases:  the  pretest, 
exploratory  trials,  and  record  trials.  The  pretest  began  in 
late  February  1990  and  lasted  until  mid  March  1990.  During 
this  time,  personnel  involved  in  the  field  test  finalized 
individual  training  on  their  weapons.  Additionally,  all  RTCA 
instrumentation  packages  were  installed  on  the  vehicles. 

A  total  of  15  exploratory  trials  were  conducted  over  an 
eight  day  period  [Ref.  1].  The  objectives  of  the  exploratory 
trials  included  allowing  maturation  of  the  data  collection  and 
reduction  procedures,  resolving  instrumentation  problems,  and 
providing  an  opportunity  for  test  controllers  and  players  to 
refine  their  procedures.  Additionally,  the  exploratory  trials 
provided  objective  evidence  about  whether  instrumentation  and 
players  were  ready  to  start  record  trials. 

Fifty  record  trials  were  conducted  from  9  April  to  23  May 
1990.  The  trials  were  conducted  in  the  same  manuever  area, 
which  had  dimensions  of  approximately  15  kilometers  by  15 
kilometers.  Each  trial  was  a  force-on-force  battle  which 
generally  lasted  about  one  hour.  Normally,  two  trials  were 
conducted  each  test  day.  The  friendly  units  were  designated 
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as  Blue  forces  and  the  'threat*  units  as  Red  forces.  The 
battles  involved  Red  and  Blue  mechanized  forces  of  battalion 
strength.  Table  I  shows  the  typical  size  and  make-up  of  Blue 
and  Red  forces  during  a  battle.  For  Red  weapon  systems, 
American  made  'surrogates'  were  used.  The  surrogates  are 
identified  in  the  table  in  parentheses. 


Table  I  BLUE  AND  RED  FORCES  IN  LOS-F-H  FIELD  TEST 


BLUE  PLAYER 

NUMBER  OF  SYSTEMS 

MlAl  TANK 

10 

M3  CAVALRY  FIGHTING  VEHICLE 

12 

AH-1  COBRA  HELICOPTER 

2 

OH- 5 8  KIOWA  HELICOPTER 

3 

A- 10  THUNDERBOLT  ATTACK  PLANE 

2 

LINE-OF-SIGHT  FORWARD  HEAVY 

4 

RED  PLAYER  (SURROGATE) 

NUMBER  OF  SYSTEMS 

FUTURE  SOVIET  TANK  (M60A3  TANK) 

10 

SOVIET  BMP  (Ml 13  ARMORED 

PERSONNEL  CARRIER) 

12 

HIND  HELICOPTER  (AH-64  APACHE) 

3 

HAVOC  HELICOPTER  (AH-64  APACHE) 

1 

HIP  HELICOPTER  (UH-60 

4 

BLACKHAWK) 

FROGFOOT  ATTACK  PLANE  (A-7 

2 

CORSAIR) 

11 


The  LOS-F-H  mission  during  all  record  trials  was  to  defend 
the  Blue  maneuver  force  against  air  attack  as  the  force 
conducted  its  mission.  The  Red  forces  had  no  comparable  Air 
Defense  Artillery  asset. 

4.  FIELD  TEST  CONDITIONS  AND  THE  SELECTION  OF  TRIALS 
The  conditions  of  the  trials  were  varied.  These 
conditions  included  north  or  south  orientation,  day  or  night, 
defense  or  offense,  and  the  Mission  Oriented  Protective 
Posture  (MOPP)  of  either  0  or  4.  MOPP  4  indicates  that  all 
players  must  wear  full  protective  chemical  garments. 

We  did  not  attempt  to  make  analyses  related  to  all  50 
field  trials.  Only  four  trials  were  analyzed,  due  to  their 
identical  conditions  of  daytime,  defense,  MOPP  0,  and  Blue  in 
the  South,  and  because  these  trials  had  the  most  helicopter 
and  fixed-wing  detection  data  available. 

B.  FIELD  TEST  RESOURCE  LIMITATIONS 

There  were  two  resource  limitations  that  could  have 
impacted  on  the  quality  of  the  data  resulting  from  the  field 
test.  One  was  the  lack  of  artillery  in  any  of  the  trials. 
The  use  of  artillery  might  have  caused  the  LOS-F-H  systems  to 
move  more  often,  thereby  reducing  their  effectiveness; 
artillery  most  likely  would  have  destroyed  several  LOS-F-H 
systems.  The  result  would  have  been  fewer  enemy  aircraft 
detected. 
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A  second  limitation  was  a  safety  precaution  that  allowed 
fixed-wing  pilots  to  fly  no  lower  than  200  feet  above  the 
ground.  The  'ground'  was  interpreted  by  the  pilots  to  be  the 
valley  floor,  and  therefore  meant  they  could  fly  closer  to 
tree-top  level  when  over  hilly  areas,  since  they  would  still 
be  200  feet  above  the  valley  floor.  This  precaution  made 
detection  of  enemy  fixed-wing  aircraft  more  likely,  since  they 
were  required  to  fly  higher  than  what  they  probably  would  fly 
in  combat.  Higher  flight  altitude  makes  the  aircraft  more 
vulnerable  to  ground  radar,  as  it  has  less  chance  of  being 
masked  by  mountains  or  other  terrain  features. 
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III.  DESCRIPTION  AND  USE  OF  JANUS  SIMULATION 

A.  OVERVIEW  OF  JANUS 

Janus  is  an  interactive,  force-on-force,  high-resolution 
combat  model  used  extensively  throughout  the  Army.  The 
original  version  of  Janus  was  developed  at  the  Conflict 
Simulation  Center  at  Lawrence  Livermore  Laboratory  for  the 
purpose  of  creating  a  two-sided  analytical  and  training  tool 
to  study  the  dynamic  battlefield.  It  was  later  improved  by 
the  Janus  Working  Group  at  Tradoc  Analysis  Command  (TRAC) , 
White  Sands  Missile  Range.  Janus  is  intended  for  use  at 
brigade-level  and  below.  The  Janus  code  is  written  in  FORTRAN 
[Ref.  4]. 

The  Janus  simulation  models  combat  systems,  such  as  tanks 
and  helicopters,  the  battlefield  environment,  and  each 
system's  interaction  with  other  systems  and  their 
environments.  The  characteristics  of  these  combat  systems 
include  descriptions  of  the  weapons  carried,  weapon 
capabilities,  and  much  more. 

B.  DEVELOPMENT  OF  THE  JANUS  TRIALS 

1.  LOADING  THE  START  POSITIONS  AND  ROUTES 

Prior  to  simulating  a  field  test  trial  in  Janus,  it  was 
necessary  to  enter  the  start  locations  and  movement  routes  of 
all  players.  These  start  locations  and  movement  routes  were 
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the  only  data  translated  directly  from  the  field  test  to  the 
Janus  trials.  This  was  done  using  a  combination  of  FORTRAN 
programs,  written  at  TRAC-Monterey  and  TRAC-White  Sands  by  Mr. 
A1  Kelner  and  CRT  Allen  East  that  rapidly  loaded  the  start 
locations  and  routes  for  each  trial. 

2.  DESCRIPTION  OF  JANUS  UTILITIES 

Before  addressing  the  specific  development  of  the  trials, 
we  describe  how  Janus  can  be  manipulated  by  the  user.  This  is 
accomplished  using  four  user-friendly  utilities  built  into  the 
simulation  software.  The  four  utilities  are  the  symbol 
editor,  the  PH/PK  (probability  hit/probability  kill)  editor, 
the  graphic  terrain  editor,  and  the  scenario  editor  [Ref.  4]. 

The  symbol  editor  allows  the  creation  or  definition  of  the 
graphic  symbols  used  by  Janus  to  mark  unit  locations.  Symbols 
may  be  drawn  as  simple  standard  military  graphics  or  as  very 
detailed  silhouettes.  The  symbol  editor  contains  many  pre¬ 
defined  symbols  that  the  user  may  select. 

The  PH/PK  editor  allows  editing  of  data  used  to  determine 
lethality  over  range  for  shooter-target  pairs.  Not  only  is 
range  varied  in  these  data  sets,  but  also  the  postures  of  the 
shooter  and  target.  For  example,  PH/PK  varies  for  a  moving  or 
stationary  shooter,  as  well  as  with  a  moving  or  stationary 
target.  A  specific  Janus  file  can  contain  up  to  1000  PH/PK 
data  sets;  each  set  represents  the  hit  and  kill  probabilities 
of  a  particular  shooter  against  a  particular  target. 
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The  graphic  terrain  editor  allows  input  and  modification 
of  terrain  features,  such  as  roads  and  vegetation,  on  a  three 
dimensional  terrain  grid.  The  terrain  resolution  used  in  the 
analysis  was  50  meters.  This  50  meter  resolution  means  that  at 
every  50  meter  interval  in  both  an  east/west  and  north/south 
direction,  a  terrain  height  reading  was  made  and  elevations 
between  adjacent  data  points  were  interpolated  to  provide  a 
relief  representation  [Ref.  5].  Janus  can  be  used  with  higher 
or  lower  terrain  resolution  as  well.  All  Janus  map  displays 
use  digitized  data  from  the  Defense  Mapping  Agency  and  land 
satellite  data. 

The  scenario  editor  provides  access  to  system/weapon 
characteristics,  battlefield  descriptive  parameters,  and 
scenario  force  structures.  Up  to  198  different  systems  can  be 
defined  and  their  characteristics  can  be  varied  in  numerous 
ways. 

3.  DEVEDOPMENT  OF  JANUS  SCENARIOS  AND  TRIALS 

Only  the  terrain  editor  and  scenario  editor  were  used 
extensively  in  Janus  scenario  development.  The  Fort  Hunter 
Liggett  terrain  representation  we  used  for  our  Janus  trials 
was  drawn  from  the  Janus  database.  The  graphic  terrain  editor 
allowed  two  improvements  in  the  terrain  representation  that 
were  based  on  personal  knowledge  of  the  area.  One 
improvement  was  the  addition  of  roads,  which  allowed  vehicles 
to  travel  along  roads  in  the  simulation  as  they  did  in  the 
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field  test.  The  other  change  was  the  addition  of  vegetation 
that  was  missed  in  the  database,  yet  was  critical  to  proper 
representation  of  the  maneuver  area. 

The  scenario  editor  was  used  extensively  to  create 
realistic  Janus  scenarios.  The  most  critical  concerns  were 
accurately  constructing  the  parameters  of  the  LOS-F-H  system 
and  recreating  the  flight  behavior  of  the  fixed-wing  aircraft 
and  helicopters.  Important  LOS-F-H  parameters  included  radar 
and  sensor  type,  maximum  detection  range,  detection 
probability  of  the  LOS-F-H  at  certain  ranges  against  various 
aircraft,  and  maximum  engagement  range.  Critical  aircraft 
parameters  were  flight  speed  and  flight  altitude.  The  values 
of  these  parameters  were  derived  from  the  field  test  reports, 
conversations  with  the  field  test  coordinator  and  two  Janus 
programmers,  and,  where  necessary,  personal  judgement  [Ref. 6]. 

C.  UNCERTAINTY  OF  PARAMETERS 

While  most  of  the  data  described  above  were  available, 
certain  information  was  not.  Most  significant  among  missing 
data  were  the  probabilities  of  detections  for  the  LOS-F-H 
against  different  types  of  aircraft.  These  values  were 
estimated  using  guidance  from  a  White  Sands  Missile  Range 
Janus  programmer  and  personal  judgement  [Ref.  6]. 
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IV.  ANALYSIS  OF  FIELD  TEST  AND  JANUS  DETECTION  DATA 


A.  REASONS  FOR  DIFFICULTY  IN  COMPARING  DETECTION  DATA 

Research  has  revealed  that  comparing  detection  data  from 
the  field  test  to  detection  data  generated  in  Janus  will  be 
difficult.  One  reason  is  the  difficulty  in  measuring 
detections  in  a  field  test.  The  Real  Time  Casualty  Assessment 
(RTCA)  system  cannot  measure  aircraft  detections  by  individual 
soldiers  manning  equipment.  Additionally,  shortcomings  in 
Janus  and  the  field  test  restrict  the  ability  to  duplicate 
the  field  test  environment.  These  problems  are  discussed  in 
detail  below. 

1.  FIELD  TEST  DETECTION  DATA  GATHERING  PROCESS 
As  described  in  Chapter  II,  the  Real  Time  Casualty 
Assessment  (RTCA)  system  was  effectively  used  to  gather  much 
data  during  the  field  test.  However,  the  RTCA  system  could 
not  provide  detection  data.  According  to  Major  Larry  Dubois, 
the  test  coordinator  from  TEC,  LOS-F-H  detection  data  was 
acquired  solely  through  analysis  of  video  camera  records. 

Each  LOS-F-H  system  had  a  video  camera  recording  the 
crew's  actions  and  words  throughout  the  trial.  The  video 
camera  was  placed  behind  the  crew  and  revealed  their  actions 
and  verbal  communications.  Personnel  doing  the  reconstruction 
selected  'target  handoff  as  an  overt  action  to  indicate  that 
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a  target  was  about  to  be  engaged.  ’Target  handoff  occurred 
when  the  radar  operator  (RO)  designates  one  of  the  targets 
under  track  to  the  gunner/electro-optics  (EO)  operator  for 
engagement.  This  could  be  verified  by  a  red  button,  visible 
on  video,  that  illuminated  when  'target  handoff  occurred.  By 
observing  the  time  into  the  trial  set  in  the  recording,  the 
range  of  the  aircraft  at  'target  handoff  was  determined  by 
the  video  reviewer  by  obtaining  the  location  of  the  LOS-F-H 
and  the  aircraft  from  the  movement  file.  Detections  were  then 
determined  by  backtracking  from  'target  handoff  to  another 
overt  action,  generally  a  verbal  cue,  alerting  the  video 
reviewer  that  a  detection  was  made  at  that  time.  The  range  of 
detection  was  then  calculated  similarly  to  'target  handoff 
range . 

This  process  has  several  problems.  First,  only  those 
detections  that  led  to  engagement,  as  indicated  by  'target 
handoff,  were  noticed.  A  significant  amount  of  data  may  have 
been  lost,  since  many  detections  likely  occurred  which  were 
never  recorded  because  the  particular  LOS-F-H  involved  may  not 
have  been  able  to  engage  each  target  it  detected. 
Additionally,  the  recorded  detection  ranges  will  have  random 
errors,  since  the  ranges  are  not  actual,  but  rather  were 
estimated  by  the  video  reviewers  who  depended  on  an  overt  sign 
rather  than  an  actual  detection.  Finally,  recorded  detection 
ranges  will  tend  to  be  shorter  than  detection  ranges  actually 
were.  As  mentioned  earlier,  detection  occurs  when  a  LOS-F-H 
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system  recognizes  a  target  and  identifies  it  as  hostile.  The 
time  of  this  event  cannot  be  determined  through  analysis  of 
the  test  data.  This  overt  sign  of  detection  occurred  at  an 
unknown  time  after  the  LOS-F-H  detected  the  target,  and  the 
approaching  target  moved  an  unknown  distance  during  this  time. 
This  problem  must  be  taken  into  consideration  before  any 
conclusions  can  be  drawn  from  detection  range  analyses. 

Another  limitation  of  the  test  range  instrumentation  is 
its  inability  to  determine  aircraft  elevation.  Knowing  the 
altitudes  of  fixed-wing  aircraft  and  helicopters  during  the 
field  test  is  important  for  accurately  representing  the  field 
test  in  a  simulation. 

2.  AIRCRAFT  FLIGHT  REPRESENTATION  IN  JANUS 

The  most  significant  problem  in  attempting  to  replicate 
the  field  test  in  Janus  had  to  do  with  representing  aircraft 
flight  characteristics.  An  actual  attacking  aircraft  has  the 
ability  to  quickly  change  altitude  and  speed  in  order  to 
confuse  enemy  air  defenses  or  to  deliver  its  ordnance  and 
quickly  leave  the  area.  A  critical  limitation  of  Janus  is 
that  each  aircraft  can  have  only  two  flight  speeds  and  two 
flight  altitudes  which  most  likely  cannot  closely  replicate 
the  actual  attack  profile  of  an  aircraft. 

Additionally,  there  are  two  basic  flight  modes  that  an 
actual  aircraft  may  use.  One  is  'constant  altitude',  in  which 
an  aircraft  remains  generally  at  a  set  altitude  above  a  fixed 
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level  on  the  ground  (often  sea  level) ,  regardless  of  changing 
terrain  elevation.  Fixed-wing  aircraft  in  the  field  test  used 
a  version  of  this  method,  flying  generally  at  a  constant  200 
feet  above  the  level  of  the  valley  floor.  The  other  flight 
mode  an  aircraft  may  use  is  called  'nap  of  earth'  or  terrain 
following.  The  altitude  of  an  aircraft  using  this  mode 
follows  the  changing  elevation  of  the  terrain.  During  the 
field  test,  helicopters  used  this  tactic. 

Only  the  'nap  of  earth'  mode  of  flight  is  available  in 
Janus,  and  therefore  maintaining  level  flight  paths  for  fixed- 
wing  aircraft  had  to  be  approximated.  This  was  done  by  using 
both  available  altitude  settings,  one  set  at  200  feet  (64 
meters)  for  any  portion  of  a  flight  path  segment  over  a  valley 
floor  and  the  other  set  at  50  feet  (16  meters)  for  a  flight 
path  segment  over  mountains.  The  height  of  50  feet  was  chosen 
since  the  average  elevation  of  the  mountainous  terrain  above 
the  valley  floor  is  150  feet. 

This  limitation  of  Janus  would  seem  to  have  great  impact 
on  fixed-wing  detection  data.  Fixed-wing  aircraft  in  Janus 
cannot  fly  steadily  at  200  feet  altitude  as  the  fixed-wing 
aircraft  generally  did  in  the  field  test.  Additionally,  the 
approximation  described  above  relies  on  an  inexact  estimation 
of  the  average  elevation  over  mountainous  terrain.  Yet  the 
results  of  a  sensitivity  analysis  indicates  that  the  effects 
of  this  estimation  may  be  insignificant. 
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A  data  sample  of  detection  ranges  containing  three 
iterations  of  one  trial  with  the  average  elevation  over 
mountainous  terrain  set  at  80  feet  (26  meters)  were  compared 
to  a  data  sample  of  detection  ranges  of  three  iterations  of 
the  same  trial  using  the  original  elevation  over  mountainous 
terrain  of  50  feet  (16  meters) .  The  method  of  comparison  was 
a  two-sided  Mann-Whitney  test.  These  two  data  samples  were 
combined  and  the  detection  ranges  were  ranked.  The  average 
ranks  of  the  two  data  samples  were  close  at  57.16  and  63.23. 
Due  to  a  sufficiently  large  sample  size  of  over  100,  the  Mann- 
Whitney  Z  statistic  is  applicable.  The  null  hypothesis  is 
that  the  distributions  of  the  two  data  sets  are  equal.  The  Z 
statistic  for  this  test  is  .949,  and  at  a  significance  level 
of  .05,  the  null  hypothesis  cannot  be  rejected.  The 
STATGRAPHICS  5.0  software  package  was  used  in  this  and  all 
other  analyses  in  this  thesis  [Ref.  7] 

Therefore,  while  the  availability  of  only  a  'nap  of  earth' 
flying  mode  in  Janus  makes  a  comparison  of  fixed-wing 
detection  data  flawed,  it  appears  that  the  approximation  used 
does  not  add  significant  additional  error. 

B.  ANALYSIS  OF  FIXED-WING  DATA 

The  fixed-wing  data  used  in  our  analysis  came  from  four 
similar  trials.  Each  trial  was  conducted  during  daylight 
hours,  had  the  Blue  forces  in  the  defense,  and  was  conducted 
in  a  chemical  and  nuclear  free  environment. 


22 


The  field  test  data  contained  at  most  one  detection  per 
pass  through  the  manuever  area  for  every  fixed-wing  aircraft. 
The  planes  always  flew  in  pairs  and  made  four  or  five  passes 
during  each  trial.  Each  trial,  therefore,  had  eight  to  ten 
detections  recorded. 

Each  set  of  Janus  data  had  some  repeated  detections.  To 
account  for  possible  dependence  between  ranges  of  repeated 
detections,  these  repeated  data  points  were  eliminated,  so 
only  the  initial  detection  ranges  remained.  No  Janus 
iteration  had  more  than  six  repeated  detections. 

A  summary  of  the  sample  sizes  of  fixed-wing  data  is 
displayed  in  Table  II  below.  The  number  in  each  box  reflects 
the  sample  size  in  each  category.  The  sample  sizes  of  Janus 
trials  are  a  result  of  five  independent  iterations  per  trial, 
while  the  field  test  data  comes  from  only  one  iteration. 


Table  II  FIXED-WING  DETECTION  SUMMARY 


TRIAL  A 

TRIAL  B 

TRIAL  C 

TRIAL  D 

TOTAL 

JANUS* 

45 

78 

79 

54 

256 

FIELD 

TEST 

10 

9 

12 

10 

41 

TOTAL 

55 

87 

91 

64 

297 

*  5  Janus  Runs  Pooled 
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A  comparison  of  the  fixed-wing  detection  data  is  displayed 
in  Figure  2.  This  series  of  box-plots  allows  comparisons  of 
the  Janus  and  field  test  data  from  each  trial  side-by-side. 
The  boxes  show  the  median  of  each  data  set  as  well  as  its 
inter-quartile  range.  This  plot  shows  no  strong  evidence  of 
a  difference  in  means  between  Janus  and  field  test  fixed-wing 
data  of  a  common  trial. 


Figure  2  Fixed-Wing  Box  Plots 
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A  Two-Factor  Analysis  of  Variance  (ANOVA)  was  used  to  test 
for  equivalence  of  means  of  detection  ranges  between  trials 
and  between  Janus  and  field  test  data  for  fixed-wing  aircraft. 
The  ANOVA  summary  table  is  shown  in  Table  III.  The  two 
factors,  or  sources  of  variation,  are  labeled  as  trials  and 
data  source,  and  are  related  to  testing  the  difference  between 
trials  and  the  difference  between  multiple  Janus  iterations 
and  the  field  test  data. 


Table  III  ANOVA  TABLE  FOR  FIXED-WING  AIRCRAFT 


SOURCE  OF 
VARIANCE 

SUM  OF 
SQUARES 

DF 

MEAN  SQUARE 

F-RATIO 

SIG 

LEVEL 

TRIALS 

97,571,767 

3 

32,523,922 

3.627 

.0135 

DATA  SOURCE 

275 

1 

275 

.000 

.9956 

INTERACTIONS 

40,502,645 

3 

13,500,882 

1.506 

.2132 

RESIDUAL 

2,596,280,000 

289 

8,967,573 

TOTAL 

2,829,300,000 

296 

STATGRAPHICS  5.0  has  two  methods  of  computing  sums  of 
squares  for  the  ANOVA  table.  The  method  we  used,  designated 
as  Type  III,  computes  the  additional  sum  of  squares  for  each 
factor  as  if  it  were  added  to  the  model  last.  This  is  the 
method  recommended  for  use  with  an  unbalanced  design  [Ref.  7]. 

Between  the  four  different  trials,  the  F-ratio  was  3.627 
(df=3,  289)  with  a  significance  level  of  .0135,  indicating  a 
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significant  difference  between  means.  Between  Janus  and  field 
test  data,  a  very  small  F-ratip  resulted  in  a  significance 
level  of  .9956,  revealing  a  strong  indication  of  no 
significantly  different  means.  The  interaction  of  the  two 
effects  is  insignificant  as  seen  by  its  F-ratio  of  1.506 
(df=3,  289)  and  significance  level  of  .2132.  However,  the 
plot  seen  in  Figure  3  below  suggests  some  possible 
interaction.  This  conflict  between  the  F-test  and 
interactions  plot  could  be  a  result  of  the  small  sample  sizes 
of  the  field  test  data.  But  the  apparent  crossover  may  be  due 
to  random  error;  the  standard  error  of  the  means  for  field 
test  data  for  each  trial  is  approximately  1000  meters,  while 
that  for  Janus  data  for  each  trial  is  approximately  375 
meters.  The  standard  error  was  calculated  by  dividing  the 
standard  deviation  of  every  field  test  or  Janus  detection  data 
set  by  the  number  of  detections  in  that  particular  data  set. 
Overall,  we  conclude  there  is  not  a  significant  interaction 
between  trials  and  source  (test  or  Janus) . 
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To  check  the  adequacy  of  the  ANOVA  model  for  fixed-wing 
aircraft  data,  the  data  were  tested  for  normality  in 
distribution  of  residuals  and  for  equality  of  variance.  The 
histogram  plot  of  residuals  versus  a  normal  curve  is  shown  in 
Figure  4.  It  indicates  an  apparent  lack  of  normality  of  the 
residuals.  A  Chi-Square  Goodness  of  Fit  Test  of  the  fixed- 
wing  data  gives  a  chi-square  statistic  of  58.47  (df=16) .  The 
corresponding  significance  level  was  .000,  verifying  that  the 
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residuals  do  not  appear  to  be  normally  distributed.  This  is 
not  a  great  concern,  due  to  the  robustness  of  the  ANOVA  with 
regard  to  normality  [Ref.  8].  Nevertheless,  the  distribution 
of  the  residuals  of  the  log  of  the  fixed-wing  data  were 
examined  and  did  not  appear  normal.  Ratios  of  sample 
variances  between  corresponding  Janus  and  field  test  trials 
and  from  trial  to  trial  resulted  in  values  close  to  one. 
Additionally,  95%  confidence  intervals  for  ratios  of  variance 
include  the  value  of  one  for  each  of  the  trials.  Therefore  a 
hypothesis  of  equality  of  variance  cannot  be  rejected. 


FIXED-WING  RESIDUALS  PLOTTED  VERSUS 
NORMAL  CURVE 


residuals  CX  'IOOD3 


Figure  4  Histogram  Of  Fixed-Wing  Residuals  Versus  Normal  Plot 
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An  additional  Two-Factor  ANOVA  was  conducted  using  the 
four  trials  versus  the  individual  Janus  runs  and  the  field 
test.  The  purpose  of  this  test  was  to  examine  the  effect  of 
the  individual  Janus  runs  and  any  resulting  residuals.  This 
was  accomplished  by  separating  the  data  source  factor  into  its 
5  Janus  and  a  field  test  level.  Table  IV  displays  the  results 
of  this  ANOVA. 

Table  IV  ANOVA  OF  FIXED-WING  DATA  BY  INDIVIDUAL  JANUS 
ITERATION 


SOURCE  OF 
VARIANCE 

SUM  OF 
SQUARES 

DF 

MEAN  SQUARE 

F-RATIO 

SIG 

LEVEL 

TRIALS 

169,800,000 

3 

56,600,498 

6.203 

.0004 

DATA  SOURCE* 

36,629,000 

5 

7,325,743 

.803 

.5485 

INTERACTIONS 

96,229,338 

15 

6,415,289 

.703 

.781 

RESIDUALS 

2,491,200,000 

273 

9,125,252 

TOTAL 

- 1 

2,829,300,000 

296 

*  5  Individual  Janus  Iterations  and  the 
Field  Test  Looked  at  Separately 


The  significance  levels  above  verify  the  previous  ANOVA, 
in  that  the  difference  in  means  between  trials  is  quite 
significant,  while  the  differences  in  means  between  Janus 
iterations  and  the  field  test  may  not  be  significant. 
Additionally,  the  ANOVA  shows  no  interactions  between  the  two 
sources  of  variance. 
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C.  ANALYSIS  OF  HELICOPTER  DETECTION  DATA 


The  helicopter  detection  data  we  considered  are  for  the 
same  four  trials  as  above  for  the  fixed-wing  case.  Each  field 
test  trial  contained  approximately  30  helicopter  detections, 
while  five  iterations  of  each  Janus  trial  resulted  in  about 
100  detections  when  pooled.  Both  sets  of  helicopter  data 
contained  repeated  detections  of  a  given  helicopter,  as  in  the 
Janus  fixed-wing  data,  and  to  avoid  dependency  among  detection 
ranges,  all  were  removed  but  the  first  of  multiple  repeated 
detections.  Table  V  below  shows  a  summary  of  the  counts  of 
helicopter  detection  ranges. 

Table  V  SUMMARY  OF  HELICOPTER  DETECTION  RANGES 


TRIAL  A 

TRIAL  B 

TRIAL  C 

TRIAL  D 

TOTAL 

JANUS* 

95 

147 

99 

98 

439 

FIELD 

TEST 

29 

40 

19 

22 

110 

TOTAL 

124 

187 

118 

120 

549 

*  5  JANUS  RUNS  POOLED 


A  graphical  comparison  of  the  helicopter  data  is  indicated 
by  the  multiple  box  plots  in  Figure  5.  The  differences 
between  Janus  and  field  test  ranges  is  obvious,  as  the  Janus 
ranges  are  larger  than  the  field  test  ranges  in  all  cases. 
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Figure  5  Comparison  of  Helicopter  Detection  Ranges 

The  helicopter  detection  data  were  compared  with  a  Two 
Factor  ANOVA.  The  ANOVA  summary  table  is  shown  in  Table  VI. 
The  two  factors  are  the  data  source  (Janus  or  field  test)  and 
the  trials.  This  ANOVA  gave  large  F-ratio  values  for  trials 
and  data  source,  indicating  that  there  are  significant 
differences  in  the  means  related  to  both  factors.  The  F-ratio 
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for  interactions  between  trials  and  data  sources  is  3.493 
(df=3,  541)  with  a  significance  level  of  .0155,  indicating 
there  is  significant  interaction  between  trials  and  data 
sources.  However,  a  plot  of  these  interactions,  seen  in 
Figure  6,  shows  the  interaction  is  due  to  only  trial  4. 

Table  VI  ANOVA  TABLE  FOR  HELICOPTER  DATA 


SOURCE  OF 
VARIATION 

SUM  OF 
SQUARES 

DF 

MEAN  SQUARE 

F-RATIO 

SIG 

LEVEL 

TRIAL 

132,660,000 

3 

44,222,000 

10.230 

.000 

DATA  SOURCE 

442,580,000 

1 

442,580,000 

102.387 

.000 

INTERACTIONS 

45,289,872 

3 

mm 

.0155 

RESIDUAL 

2,338,500,000 

541 

4,322,579 

TOTAL 

3,242,600,000 

548 
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The  adequacy  of  the  Two-Factor  ANOVA  was  checked  by 
testing  whether  the  residuals  have  a  normal  distribution  and 
whether  the  detection  ranges  have  common  variance.  Figure  7 
shows  a  araphical  comparison  of  residuals  to  a  normal  curve. 
It  suggests  the  residuals  are  not  normally  distributed.  A 
Chi-Square  Goodness  of  Fit  Test  verified  this  lack  of 
normality,  as  the  chi-square  statistic  is  87.93  (df=19)  and 
has  a  significance  level  of  .000.  However,  the  robustness  of 
the  ANOVA  test  regarding  lack  of  normality  in  the  residuals 
makes  it  still  valid  but  not  as  powerful  [Ref  8]. 
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RES  I  DUALS 


Figure  7  Helicopter  Residuals  Versus  Normal  Plot 

The  ratios  of  variances  between  each  set  of  corresponding 
Janus  and  field  test  data  indicate  common  variance  per  trial. 
Additionally,  scatterplots  of  detection  means  versus  standard 
deviations  for  each  set  of  helicopter  data,  shown  in  Figure  8, 
indicate  the  similarity  of  standard  deviations  and  the  lack  of 
any  strong  relationship  between  the  mean  and  standard 
deviation.  Therefore,  the  assumption  of  common  variance 
appears  valid. 
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An  additional  Two-Factor  ANOVA  was  again  run  with  the 
helicopter  detection  data  using  trials  as  one  factor  and  data 
sources  as  the  other  factor.  For  this  test,  however,  data 
source  is  split  into  the  five  levels  corresponding  to  Janus 
iterations  per  trial;  the  sixth  level  of  this  factor  is  the 
field  test.  The  resulting  ANOVA  summary  is  shown  in  Table 
VII. 
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Table  VII  ANOVA  TABLE  OF  HELICOPTER  DETECTION  DATA 


SOURCE  OF 
VARIATION 

SUM  OF 
SQUARES 

DF 

MEAN 

SQUARES 

F-RATIO 

SIG 

LEVEL 

TRIALS 

358,750,000 

3 

119,580,000 

27.31 

.000 

DATA  SOURCE* 

452,840,000 

5 

90,567,000 

20.68 

.000 

INTERACTION 

69,738,112 

15 

4,649,207 

1.06 

.389 

RESIDUALS 

2,298,800,000 

525 

4,378,711 

TOTAL 

3,242,600,000 

548 

*  5  INDIVIDUAL  JANUS  ITERATIONS  AND 
FIELD  TEST  EVALUATED  SEPARATELY 


Trials  and  data  source  factors  are  again  highly 
significant.  However,  the  interaction  between  trials  and  data 
sources  is  insignificant  in  this  case,  in  contrast  with  the 
model  used  in  connection  with  Table  VI.  This  suggests  that 
when  the  variability  among  Janus  runs  is  accounted  for,  the 
apparent  interactions  become  insignificant.  The  increase  in 
interaction  sum  of  squares  in  Table  VII  is  more  than  offset  by 
the  increase  in  degrees  of  freedom. 


t 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  analyses  of  a  limited  set  of  data  show  that  that  Janus 
and  field  test  aircraft  detection  ranges  do  not  agree.  Our 
conclusion  is,  therefore,  that  we  are  not  able  to  validate  or 
accredit  Janus  for  use  with  Model-Test-Model  at  this  time. 
This  is  due  in  part  to  the  difficulty  in  replicating  the  field 
test  process  in  Janus. 

The  comparison  of  fixed-wing  data  showed  differences  in 
the  means  of  detection  ranges  between  trials,  but  no 
significant  difference  between  sources.  Yet,  the  meaning  of 
this  result  is  not  clear,  since  both  sets  of  data  being 
compared  were  limited.  Additionally,  the  powers  of  the  tests 
are  low  due  to  high  variance  and  low  field  test  sample  sizes. 

B.  RECOMMENDATIONS 

We  have  found  that  an  attempt  to  replicate  a  field  test 
trial  in  a  simulation  such  as  Janus  requires  a  thorough 
knowledge  of  the  field  test  by  the  modeler.  Detailed 
information  such  as  flight  altitudes,  speeds,  movement  routes 
used  by  aircraft  and  vehicles  on  the  ground,  and  other 
characteristics  not  currently  provided  in  field  test  data  are 
needed.  To  ensure  this  is  done,  modelers  and  testers  should 
closely  coordinate  prior  to  and  throughout  the  field  test. 
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Additionally,  setting  correct  parameters  within  Janus  is 
critical  to  the  accurate  portrayal  of  the  weapon  systems 
involved.  The  Army  proponent  is  the  most  knowledgeable  source 
of  weapon  capabilities,  and  should  therefore  work  closely  with 
the  modellers  prior  to  the  pre-test  to  ensure  the  parameters 

4 

are  set  correctly. 

Another  important  finding  is  that  detections  are  too 
difficult  to  measure  in  the  LOS-F-H  field  test  by  the  use  of 
video  data  reduction.  Other  battlefield  occurrences,  such  as 
engagements  and  kills,  can  be  recorded  in  'real  time*  in  a 
field  test  environment.  Future  comparisons  should  use  a 
phenomenon  other  than  detections  that  is  well  defined  and  is 
recorded  by  the  instrumentation. 

Finally,  the  topic  of  MTM  deserves  further  investigation. 

Future  field  tests  of  the  M1A2  battle  tank  have  already 
benefitted  from  the  pre-test  phase  of  MTM,  and  the  data 
resulting  from  the  upcoming  M1A2  field  tests  should  be 
analyzed  and  compared  with  Janus  data,  with  the  intent  of 
validation  or  accreditation  of  Janus. 


« 
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